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Abstract 

We  present  a  seasonal  state-space  model  using  Kalman  re¬ 
cursions  to  learn  and  predict  structured  behavior  patterns. 
The  model  is  employed  to  detect  events  using  the  learned 
expectations  of  typical  scene  activity.  We  demonstrate  the 
approach  for  modeling  the  expected  number  of  pedestrians 
in  a  scene  over  hour-long  periods  (over  multiple  days)  and 
for  detecting  event  anomalies.  The  framework  provides  a 
single  long-term  model  by  exploiting  the  natural  seasonal 
trends  in  daily  human  activity. 

1.  Introduction 

An  important  aspect  of  any  intelligent  surveillance  and 
monitoring  system,  beyond  person  detection  and  activity 
recognition,  is  the  ability  to  automatically  learn/discover 
patterns  or  trends  of  scene  behavior  and  to  identify  anoma¬ 
lous  events.  For  example,  one  may  wish  to  be  alerted  when 
a  person  is  present  in  a  secure  unpopulated  area,  if  a  per¬ 
son  is  seen  at  a  particular  time  of  day  when  there  should 
be  no  people  present  (e.g.,  late  at  night),  or  if  an  individ¬ 
ual  remains  in  an  area  for  an  extended  time.  We  also  may 
be  interested  in  more  dynamic  events  such  as  if  a  person  is 
seen  following  an  unusual  path  or  running  across  a  normally 
quiescent  area. 

As  most  video  cameras  are  positioned  to  monitor  a  fixed 
area  over  long  periods  of  time  (even  several  years),  it  is  not 
unreasonable  to  insist  that  the  system  should  learn  the  ac¬ 
tivity  trends  by  observation  and  adapt  itself  through  expe¬ 
rience  (modifying  its  model  parameters).  Obviously,  cer¬ 
tain  types  of  event  alerts  will  require  manually-created  de¬ 
tectors,  but  statistical  modeling  of  activity  trends  over  long 
periods  can  be  employed  to  automatically  derive  expecta¬ 
tions  of  scene  behaviors.  Then,  any  observed  behavior  that 
violates  the  scene  expectation  can  be  considered  to  be  an 
anomalous  event.  The  main  advantage  of  a  statistical  ap¬ 
proach  is  that  individual  detectors  for  all  possible  atypical 
events  do  not  need  to  be  hand-crafted.  Specifics  of  the  event 


(e.g.,  location,  action,  time-of-day)  could  also  be  used  to 
classify  the  anomaly  for  video  storage  and  retrieval  pur¬ 
poses  (particularly  helpful  to  a  video  analyst). 

Due  to  our  highly  scheduled  lifestyles,  human  activity 
tends  to  exhibit  the  pervasive  phenomenon  of  “seasonality”, 
referring  to  the  tendency  to  repeat  a  high-and-low  occur¬ 
rence  patterns  of  behavior  across  the  day/week.  For  exam¬ 
ple,  on  a  college  campus  many  people  will  be  seen  between 
class  periods  and  few  during  the  class  times  (except  for  the 
occasional  latecomers).  In  Fig.  1,  we  show  a  plot  of  the 
number  of  people  present  in  a  particular  scene  on  a  campus 
from  9- 10am  (sampled  every  15  seconds)  on  three  Mon¬ 
days.  Though  there  is  a  fairly  large  variation  at  any  partic¬ 
ular  time  across  the  three  days,  there  does  exist  an  overall 
rise-and-fall  pattern  for  the  presence  of  the  people  across 
the  hour  (the  peak  occurs  at  the  time  between  classes,  9:20- 
9:30am).  Such  time-structured  patterns  are  common  in  both 
public  and  private  spaces.  A  method  that  can  exploit  natu¬ 
ral  seasonal  activity  trends  will  be  better-suited  to  represent 
and  predict  the  behavior  patterns  for  anomaly  detection. 

In  this  paper,  we  present  a  seasonal  state-space  model 
for  learning,  predicting,  and  validating  expectations  of  typ¬ 
ical  scene  activity.  In  particular,  we  focus  on  modeling  the 
presence  patterns  (counts)  of  pedestrians  in  hour-long  time 
periods  on  particular  days  of  the  week  on  a  college  cam¬ 
pus.  The  approach  is  based  on  a  seasonal  time  series  model 
applied  over  hourly  periods,  rather  than  using  a  brute-force 
method  with  a  separate  model  for  each  second  of  each  day. 
The  state  and  noise  parameters  for  the  seasonal  model  are 
initialized  from  training  data,  and  Kalman  recursions  are 
employed  to  optimally  estimate  the  state  sequence  and  to 
forecast  the  next  time  period.  We  detect  an  anomalous  event 
when  a  new  observation  exceeds  the  confidence  range  for 
the  predicted  (expected)  behavior.  The  model  is  updated 
with  new  observations  to  adapt  to  slowly  changing  trends 
and  seasonality.  We  demonstrate  the  approach  on  synthetic 
data  and  thermal  video  surveillance  imagery. 

The  remainder  of  this  paper  is  described  as  follows.  We 
begin  with  a  review  of  related  work  (Sect.  2).  Next  we 
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Figure  1 :  Number  of  people  manually  counted  in  a  particu¬ 
lar  scene  from  9-10am  (every  15  sec.)  on  three  Mondays. 

describe  the  background  and  formulation  for  the  proposed 
seasonal  framework  (Sect.  3).  We  then  present  experimen¬ 
tal  results  (Sect.  4).  Lastly,  we  conclude  with  a  summary  of 
the  research  and  describe  future  work  (Sect.  5). 

2.  Related  Work 

To  model  activity  patterns  and  detect  anomalous  events, 
several  graph-based  and  trajectory-based  approaches  have 
been  proposed.  A  Dynamic  Bayesian  Network  was  em¬ 
ployed  in  [10]  to  model  temporally-correlated  events  for 
detecting  typical  and  atypical  behaviors.  In  [8],  the  trajec¬ 
tories  of  people  were  used  to  build  a  semantic  scene  model 
(entry  zones,  paths,  junctions)  from  which  activity  expecta¬ 
tions  could  be  extracted.  In  [6],  a  statistical  model  of  pedes¬ 
trian  trajectories  was  used  to  learn  typical  paths  and  to  iden¬ 
tify  incidents  of  unexpected  behavior.  A  polygonal  shape 
configuration  and  its  deformation  for  path  behaviors  of  peo¬ 
ple  were  examined  for  abnormal  changes  in  [9].  Anomalous 
office  activities  were  detected  in  [1]  by  identifying  inputs 
having  a  low  likelihood  to  an  entropically-estimated  HMM. 

Aggregate  trajectory-based  methods  can  be  used  to  learn 
interesting  path  models,  but  these  methods  generally  ignore 
how  the  occurrence  of  these  behaviors  change  with  respect 
to  time  (e.g.,  over  hours  or  days).  Our  proposed  approach 
exploits  the  natural  seasonality  of  behavior  to  learn  proper 
time-based  expectations  for  detecting  event  anomalies. 

3.  Seasonal  Forecasting 

The  classical  decomposition  model  for  a  seasonal  time  se¬ 
ries  is  an  explicit  representation  composed  of  the  underly¬ 
ing  trend ,  seasonal  variation,  and  irregular  (random)  noise 
components  [2].  Other  more  generic  techniques  include 


autoregressive  moving  average  models  (ARMA,  ARIMA, 
SARIMA).  An  advantage  to  the  classical  decomposition 
model  is  that  we  have  a  representation  for  an  underlying 
process  model  of  the  time  series. 

The  classical  decomposition  model  for  a  univariate  ob¬ 
servation  y  at  time  t  is  given  by 

yt=mt  +  st  +  nt  (1) 

where  rrit  is  the  trend  component,  st  is  the  seasonal  compo¬ 
nent,  and  nt  is  noise.  The  seasonal  component  st  has  a  pe¬ 
riod  of  d  with  the  properties  St+d  =  Sf  and  Yli= o  st+i  =  0 
(i.e.,  zero  mean).  In  Fig.  1,  the  trend  mt  is  a  constant 
value  (~  4)  with  the  seasonal  values  having  a  unimodal  rise- 
and-fall  pattern  over  the  period  (with  the  addition  of  noise). 
Once  the  three  components  have  been  estimated  (from  train¬ 
ing  data),  the  properties  of  the  time  series  can  be  used  to 
predict  a  future  observation  y/f+r,  (n-step  prediction)  by  run¬ 
ning  the  model  forward  in  time.  This  will  become  useful  for 
detecting  anomalies. 

3.1.  Seasonal  Kalman  Filter  (SKF) 

A  state-space  representation  of  the  classical  decomposition 
model  can  be  used  to  avoid  the  deterministic  strictness  of 
the  components  by  allowing  the  trend  and  seasonal  compo¬ 
nents  to  evolve  randomly  in  a  recursive  manner  [2].  A  state- 
space  model  for  a  time  series  Y t  (potentially  multivariate) 
consists  of  two  fundamental  equations: 

Yt  =  GXt  +  Wt  (2) 

Xt+1  =  HXt  +  Vt  (3) 

The  obseri’ation  equation  (Eqn.  2)  gives  Yt  as  a  linear 
function  of  the  state  variable  Xt  plus  Gaussian  measure¬ 
ment/observation  noise  Wj,  and  the  state  equation  (Eqn. 
3)  determines  the  next  state  Xt+i  from  a  linear  function  of 
the  current  state  Xt  plus  Gaussian  process  noise  Vt.  Typi¬ 
cally,  the  noise  is  treated  as  independent  of  time  (dropping 
the  subscript  t). 

The  univariate  decomposition  model  (Eqn.  1)  can  be  for¬ 
mulated  in  a  recursive  state-space  model  (here  with  Yt  = 
yt)  as  follows.  First,  the  4-dimensional  state  Xt  is  set  to 

Xt  =  [  mt  st  st- 1  •  •  •  St-d+2  ]  (4) 

consisting  of  the  trend  and  the  ( d  —  1)  most-recent  season¬ 
ality  values.  The  corresponding  observation  and  state  equa¬ 
tions  are  given  by 
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For  the  observation  equation  (using  G  and  W),  the  ma¬ 
trix  G  adds  together  the  trend  rnt  and  the  current  seasonality 
value  St,  followed  by  the  inclusion  of  the  observation  noise 
in  W.  For  the  state  equation  (using  H  and  V),  the  matrix 
H  (when  applied  to  Xt)  simply  retains  the  trend  mt,  but 
updates  the  seasonality  st  using  the  zero-mean  constraint 
with  the  most-recent  ( d  —  1)  seasonality  values;  the  most- 
recent  ( d  —  2)  seasonality  values  are  shifted  along  through 
time.  We  only  add  process  noise  (V)  to  the  two  initial  state 
elements  ( mt ,  St),  as  the  remaining  terms  are  only  shifted 
seasonality  values. 

Given  values  for  the  initial  state  Xo,  the  noise 
(co)variances,  and  a  sequence  of  observations  Y°bs,  stan¬ 
dard  Kalman  recursions  can  be  employed  to  optimally  esti¬ 
mate  the  Xt  state  sequence  [7].  From  the  state  sequence,  the 
predicted  observations  are  given  by  Y  fred  =  G'X, .  As  the 
observed  data  are  continually  incorporated  into  the  model 
over  time  (using  either  Kalman  filtering  or  smoothing  re¬ 
cursions),  the  model  adapts  to  slowly  changing  trends  and 
seasonality. 

3.2.  Anomaly  Detection 

The  Kalman  recursions  for  the  state-space  model  also  prop¬ 
agate  an  error  covariance  Pt  for  each  state  Xt.  When  we 
encounter  a  new  observation,  we  can  use  the  estimated 
state  error  covariance  as  a  means  to  set  the  confidence 
levels/bounds  (e.g.,  90%  confidence)  for  detecting  outliers 
(events)  from  the  predicted  observation. 

To  transform  the  state  error  covariance  Pt  into  a 
(co)variance  measurement  in  the  observation  space,  we  use 

Ct  =  GPtGT  +  R  (9) 

with  the  observation  noise  covariance  R  =  E[WW  1  |.  We 
can  then  use  the  Mahalanobis  distance  to  determine  the  sta¬ 
tistical  match  of  the  actual  observation  Y  °bs  to  the  model 
prediction  Ybt  ed.  The  general  (multivariate)  Mahalanobis 
distance  is  given  by 

D  =  \J (Yfs  -  Yptred)T Ctx{ Yfs  -  Yptred)  (10) 

The  distance  D  is  given  in  standard  deviations,  therefore  we 
can  threshold  the  distance  with  statistical  confidence  (e.g., 
>3  SD  is  a  statistical  outlier). 


The  distance  calculation  could  be  applied  to  each  new 
observation  Y°bs  to  detect  anomalous  events,  followed  by 
an  immediate  update  of  the  model  (given  the  new  observa¬ 
tion).  However,  since  we  are  employing  seasonality  con¬ 
straints  within  the  model,  we  instead  employ  a  seasonal 
forecast  of  the  entire  period  and  use  the  state  error  covari¬ 
ance  P  computed  for  the  first  time  step  for  the  entire  period. 
We  only  update  the  model  and  P  after  the  entire  period  has 
been  tested. 

This  method  is  used  for  two  reasons.  First,  the  model  up¬ 
date  with  Kalman  recursions  (including  matrix  inversions) 
is  computationally  taxing  when  the  length  of  the  period  is 
large.  For  the  data  shown  in  Fig.  1,  the  period  is  240  time 
steps.  Thus  Xt  is  of  size  240  x  1  ( II  and  G  are  similarly 
large).  In  our  approach,  we  forecast  the  entire  period  before 
the  period  begins,  which  can  be  computed  on  an  auxiliary 
processor.  Second,  if  we  blindly  incorporate  severe  outliers 
into  the  model  as  they  are  encountered  (true  anomalies  be¬ 
yond  the  noise  expectations),  the  model  can  be  altered  and 
the  remaining  predictions  for  the  period  can  be  adversely 
affected.  We  instead  incorporate  the  observations  into  the 
model  only  after  the  entire  period  has  been  examined,  and 
also  limit  the  influence  of  any  outliers  to  ±3  SD  of  the  ex¬ 
pected  value  (to  allow  only  slow  changes  over  time). 

3.3.  Extension  for  SKF  Modeling  of  Pedestrian 
Presence 

The  above  univariate  SKF  formulation  is  well-suited  to  sea¬ 
sonal  patterns,  however  there  is  a  problem  if  it  is  applied 
directly  to  the  particular  class  of  seasonal  data  as  shown  in 
Fig.  1.  In  that  data,  there  is  a  fairly  large  variation  of  pedes¬ 
trian  counts  at  each  time  step  (across  the  three  days).  Thus, 
after  initialization  and  training,  a  new  sequence  of  obser¬ 
vations  need  only  be  individually  within  the  wide  tolerance 
at  each  time  step  to  be  considered  “typical”  and  thus  no 
strict  consideration  of  the  overall  pattern  for  the  period  is 
enforced.  This  would  certainly  cause  a  problem  if  we  took 
the  brute-force  approach  of  using  an  individual  Kalman  fil¬ 
ter  for  each  time-step  (for  each  day). 

Consider  the  case  for  a  new  period  of  observations  with 
a  low  and  constant  pedestrian  count  of  0-3  people.  In  re¬ 
lation  to  Fig.  1,  at  best  only  a  few  outliers  near  the  bump 
in  the  time  period  will  be  detected  as  atypical.  Individually, 
most  observations  may  be  within  the  wide  bounds  of  the 
model,  but  collectively  there  is  no  rise-and-fall  seasonality 
as  expected.  One  may  be  tempted  to  first  smooth  the  pedes¬ 
trian  counts  to  reduce  this  variation  and  then  model  these 
smooth  patterns  with  the  SKF.  But  unfortunately  this  would 
smooth-out  and  ignore  any  brief,  yet  meaningful,  spikes  or 
vacancies  in  pedestrian  counts.  Such  events  do  in  fact  occur 
and  could  be  especially  prevalent  in  time-lapse  monitoring. 

Our  approach  to  handle  this  problem  is  to  employ  two 
SKF  models  to  account  for  the  raw  and  locally-smoothed 
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pedestrian  counts.  Thus  we  can  better  model  the  sea¬ 
sonality  with  the  locally-smoothed  counts  yet  still  detect 
spikes/vacancies  using  the  raw  counts.  If  we  employed 
a  simple  linear  smoothing  filter  (e.g.,  mean/average),  we 
could  easily  incorporate  it  into  a  single  unified  SKF  model, 
rather  than  employing  two  independent  models.  However, 
such  a  linear  smoothing  filter  will  be  adversely  affected 
by  event  spikes  (corrupting  values  in  the  local  area  around 
the  events).  Due  to  the  brief  and  prominent  nature  of  out¬ 
liers/spikes,  we  found  that  the  use  of  a  median  filter  is  more 
appropriate  than  a  mean  or  Gaussian.  The  median  will  not 
be  corrupted  by  the  brief  event  spikes.  In  our  domain  of 
pedestrian  counts  (sampled  every  15  sec.),  we  found  that  a 
12-tap  causal  median  filter  was  appropriate  to  filter  out  short 
event  bursts  (it  typically  takes  less  than  2  minutes  for  people 
to  traverse  the  scene). 

Our  new  method  is  simply  to  run  a  raw-input  SKF  in 
parallel  with  a  median-filtered  SFK.  If  an  outlier  is  found 
(using  Eqn.  10)  in  either  of  the  models,  then  an  event  is 
registered.  Hence,  we  can  now  detect  precise  event  spikes 
using  the  raw  data  (without  corrupting  surrounding  time 
steps)  and  ensure  longer-term  behavior  patterns  for  the  pe¬ 
riod  (e.g.,  general  rise-and-fall  patterns)  using  the  median 
data. 

4.  Experiments 

To  evaluate  the  proposed  SKF  framework,  we  tested  the 
approach  for  modeling  pedestrian  counts  and  detecting 
anomalies  using  synthetic  data  and  real  thermal  imagery 
with  a  pedestrian  detector  algorithm. 

4.0.1.  Model  Initialization 

We  begin  with  an  initialization  of  the  state  estimate  Xq  and 
the  variances  for  the  noises  W  and  V.  To  give  a  reasonable 
estimate  of  Xo,  we  calculate  the  mean  for  the  first  period  of 
the  training  data  (detector  results)  and  assign  it  to  rrit,  and 
then  use  the  most  recent  ( d  —  1)  mean-subtracted  data  val¬ 
ues  as  the  seasonality  estimates  for  St.-d+  2  through  .st.  The 
initial  state  error  is  set  large  to  P0  =  105/  and  is  reduced 
during  training. 

To  estimate  the  noise  parameters  (from  the  real  data), 
we  compute  the  measurement  noise  (co)variance  R  = 
/i'[WW  1  |  from  training  data  using  the  differences  between 
the  manually-counted  people  and  the  automatic  detector  re¬ 
sults  per  frame  (for  all  training  images).  This  is  done  in¬ 
dependently  for  each  model  (raw,  median).  Similarly,  the 
process  noise  covariance  Q  =  R\W  1  ]  is  estimated  from 
training  data  for  each  model  (raw,  median)  using  the  period- 
long  mean  trend  differences  and  the  differences  between  the 
mean-subtracted  count  differences  at  each  time  step  (across 
multiple  periods).  Here  we  use  the  manual  counts  to  esti¬ 
mate  the  process  noise,  as  they  are  not  corrupted  by  mea- 


Figure  2:  One  period  of  synthetic  data  (raw  and  median- 
filtered  values). 

surement  noise.  We  note  that  an  alternate  maximum  likeli¬ 
hood  noise  estimation  process  could  also  be  employed  [2], 

4.1.  Synthetic  Data 

We  created  a  set  of  synthetic  data  similar  to  the  sequences 
shown  in  Fig.  1.  We  first  employed  a  Gaussian  rise-and- 
fall  pattern  stretched  over  240  samples  (cr  =  24).  Next  the 
pattern  was  shifted  up  by  a  constant  trend  value  (~  4).  To 
incorporate  the  noise,  we  began  by  substituting  Eqn.  3  into 
Eqn.  2  (assuming  W  and  V  are  uncorrelated)  to  yield 

Y  t  =  GHXt-1+u>  (11) 

where  to  =  GV  +  W  and  Q  =  E[lout]  =  GQGT  +  R. 
We  used  the  Q  and  R  (co)variances  estimated  from  three 
Monday  training  days  using  the  method  provided  above  in 
Sect.  4.0.1.  The  noise  to  was  added  to  the  raw  synthetic 
data  and  the  corresponding  median  data  was  formed  using 
a  12-tap  causal  median  filter.  An  example  of  one  synthetic 
period  (raw,  median)  is  presented  in  Fig.  2. 

We  trained  the  SKF  models  using  5  synthetic  periods  and 
then  analyzed  the  model  on  new  synthetic  test  data.  First  we 
tested  a  new  period  generated  using  the  same  parameters 
employed  to  create  the  synthetic  training  data.  As  expected 
the  model  closely  matched  the  new  test  data.  The  largest 
deviation  found  (using  Eqn.  10)  in  the  test  period  was  2.7 
SD.  Next  we  applied  4  event  spikes  (using  a  constant  ±10 
SD  of  the  noise  w)  to  a  new  test  period,  as  shown  in  Fig.  3 
(top  plot).  Using  a  threshold  of  3  SD  in  Eqn.  10,  the  4  event 
spikes  (and  only  those  4)  were  detected  as  anomalies.  The 
median-filtered  values  were  unaffected  (as  desired)  by  the 
events,  and  thus  the  events  were  detected  solely  in  the  raw 
count  model.  Lastly,  we  tested  a  uniform  pedestrian  count 
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Figure  3:  Synthetic  event  results.  Top  plot:  Event  spikes. 
Bottom  plot:  Inactivity.  Events  are  marked  with  ‘o’. 

of  0  (inactivity)  over  the  240  samples.  The  results  are  shown 
in  Fig.  3  (bottom  plot),  in  which  52  samples  were  identified 
as  outliers  (using  the  same  3  SD  threshold  as  before).  In 
this  example,  all  of  the  outliers  were  detected  in  the  median 
model,  but  a  few  (4)  of  the  events  were  also  found  in  the 
raw  model. 

We  also  added  several  normal  periods  after  the  inactiv¬ 
ity  period  to  determine  how  long  the  model  would  take  to 
recover  from  the  outliers  after  using  the  constrained  update 
process  (as  explained  at  the  end  of  Sect.  3.2).  The  model 
required  2  periods  of  normal  activity  (after  the  inactivity  pe¬ 
riod)  to  stabilize  and  report  absolutely  no  events. 

4.2.  Thermal  Surveillance  Imagery 

To  test  the  approach  with  real  data,  we  recorded  several 
thermal  video  sequences  of  pedestrian  traffic  on  a  college 
campus  from  9-10am  (1  frame  captured  every  15  sec.)  on 
several  Mondays,  Wednesdays,  and  Fridays  at  a  particular 
location  (15  days,  3600  frames  total).  The  number  of  pedes¬ 
trians  in  each  image  were  manually  counted  for  ground 
truth  comparison.  The  manual  counts  for  three  Mondays 
are  shown  in  Fig.  1. 

To  automatically  detect  the  pedestrians  in  each  image, 
we  employed  the  two-stage  template-based  method  of  [3]. 
The  approach  initially  performs  a  fast  screening  procedure 
using  a  generalized  thermal  contour  template  [5,  4]  to  lo¬ 
cate  potential  pedestrian  locations.  Next  an  AdaBoosted  en¬ 
semble  classifier  using  automatically  tuned  filters/classifiers 
is  employed  to  test  the  hypothesized  pedestrian  locations. 
This  method  was  trained  using  a  sample  of  1 14  frames  from 
the  thermal  video  collection,  and  then  the  detector  was  used 
to  count  the  number  of  pedestrians  in  all  of  the  images. 
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Figure  4:  Pedestrian  detection  results  in  thermal  imagery. 


Example  thermal  images  showing  the  detection  results  are 
shown  in  Fig.  4. 

We  assigned  a  separate  raw/median  SKF  model  to  each 
day  of  the  week  (e.g.,  only  for  Mondays).  Though  the  over¬ 
all  pattern  in  each  day  of  the  week  was  somewhat  similar, 
we  found  that  Fridays  during  that  time  period  were  dif¬ 
ferent  enough  from  Mondays  and  Wednesdays  to  warrant 
a  separate  model  (fewer  people  were  present  in  the  scene 
on  Fridays).  We  trained  the  models  (for  each  day  of  the 
week)  using  3  manually-counted  days.  The  noise  covari¬ 
ances  Q  and  R  were  estimated  from  the  training  data,  and 
a  12-tap  causal  median  filter  was  used  for  smoothing.  To 
set  the  detection  threshold  for  a  raw  (or  median)  model,  we 
tested  each  training  day  in  parallel  and  selected  an  outlier 
threshold  such  that  no  events  occurred  on  any  of  its  train¬ 
ing  data.  The  event  thresholds  (raw/median)  were  2.0/1. 5, 
4. 5/1.0,  and  3. 0/1.0  SD  for  the  Monday,  Wednesday,  and 
Friday  models,  respectively. 

We  show  the  testing  results  on  two  new  days  for  each 
model  in  Fig.  5.  As  described  in  Sect.  3.2,  the  models  are 
updated  after  each  period.  In  the  Monday  sequence  results, 
1  anomaly  was  detected  in  the  median-filtered  data  (the  first 
event)  and  3  anomalies  were  found  in  the  raw  counts.  We 
show  the  raw  data  with  the  events  labeled  in  the  top  plot 
of  Fig.  5.  Upon  inspection  of  the  data,  we  feel  these  de¬ 
tections  were  reasonable.  For  the  Wednesday  results,  we 
found  a  short  cluster  of  detections  (12  time  steps  long)  only 
in  the  median  model.  In  the  middle  plot  of  Fig.  5,  we  show 
the  median  data  with  the  detected  outliers.  It  is  clear  that 
there  was  a  significantly  lower  presence  of  people  during 
that  short  interval.  Lastly,  in  the  bottom  plot  of  Fig.  5,  we 
detected  no  events  in  the  Friday  sequences. 

We  additionally  were  able  to  record  a  Monday  on  a  hol¬ 
iday  showing  few  pedestrians  in  the  scene.  The  detected 
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Figure  5:  Testing  results  for  2  periods  of  real  data.  Top  plot:  Mondays  (raw).  Middle  plot:  Wednesdays  (median).  Bottom 
plot:  Fridays  (median). 


events  (61  anomalies)  are  shown  in  Fig.  6  (top  plot).  Events 
were  found  in  both  the  raw  and  median  models  (the  median 
model  detected  every  event).  Since  we  had  no  day  avail¬ 
able  with  truly  known  event  spikes,  we  simulated  a  large 
pedestrian  count  near  the  end  of  the  period  (for  1  min.)  for 
one  of  the  Monday  training  days  (simulating  a  tour  group 
passing  through  the  scene).  The  detection  result  targeting 
the  event  is  shown  in  Fig.  6  (bottom  plot).  As  in  the  previ¬ 
ous  synthetic  case,  the  model  was  able  to  target  the  events 
specifically  without  giving  false  positives  for  surrounding 
valid  presence  counts. 

5.  Summary  and  Conclusions 

We  presented  a  seasonal  state-based  model  for  learning, 
predicting,  and  validating  longer-term  behavioral  trends. 
The  approach  is  based  on  a  model  containing  the  trend,  sea¬ 
sonal  variation,  and  noise  over  a  time  period  (e.g.,  hour), 
rather  than  employing  a  brute-force  method  with  a  separate 


model  for  each  second  of  each  day.  Kalman  recursions  are 
employed  to  optimally  estimate  the  state  sequence  for  given 
set  of  observations.  Predictions  for  the  following  period 
and  their  confidences  are  used  to  detect  deviations  (event 
anomalies)  from  the  expected  behavior. 

We  demonstrated  the  framework  for  the  task  of  model¬ 
ing  the  presence  patterns  of  pedestrians  on  a  campus  during 
particular  days  of  the  week.  We  initially  showed  the  per¬ 
formance  of  the  model  on  synthetic  examples.  Then  we 
demonstrated  the  model  on  actual  pedestrian  counts  pro¬ 
vided  by  a  person  detection  algorithm  applied  to  a  collec¬ 
tion  of  thermal  surveillance  imagery.  The  results  showed 
that  the  model  is  well-grounded  in  the  seasonal  nature  of 
the  data  and  that  it  is  capable  of  reliably  detecting  events. 

In  future  work,  we  will  extend  the  approach  to  include 
action  recognition  labels  in  a  multivariate  prediction  of  sea¬ 
sonal  presence  and  action.  We  will  also  collect  a  video 
database  of  hour-long  periods  during  the  day  and  across 
months  to  evaluate  the  longer-term  applicability  of  the  ap- 


6 


30 


_<U 

Q. 

O 

<U 

CL 

=tt 


-10 


1  1  1  1  1  1  1  1  1 

^  1 

—  yt 

prediction 

O  outliers 

'  i 

/h'O/, 

h  ^  /v  / 

y  t  \l  \  ‘\ 

\  1  r  r  t  \l  *\i  ;  \A\t  1 

_ i _ i _ l _ i _ l _ l _ i _ l _ i _ i _ l _ 

20  40  60  80  100  120  140  160  180  200  220  240 

Time  Step 


Time  Step 


Figure  6:  Anomalies.  Top  plot:  Holiday  inactivity.  Bottom  plot:  Simulated  event  spikes. 


proach.  Furthermore,  we  plan  to  use  prior  event  knowledge, 
such  as  a  calendar  with  scheduled  holidays,  to  further  con¬ 
strain  the  model  updates  when  anomalies  are  encountered. 
As  seasonal  behavior  patterns  are  quite  common  in  our  so¬ 
ciety,  we  expect  this  model  to  be  useful  for  further  study  in 
relation  to  automatic  surveillance  and  monitoring  systems. 
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